Imaging apparatus, method of processing captured image, and program for processing captured image

ABSTRACT

Two kinds of color images, obtained by a solid-state imaging device, having different color tones are combined with each other. A first captured color image due to a first pixel group (pixels in which spectral sensitivities of color filters are wide) of the solid-state imaging device is processed. A second captured color image due to a second pixel group (pixels in which spectral sensitivities of color filters are narrow) is processed. The level difference between captured image signals of the pixels of the first pixel group and captured image signals of the pixels of the second pixel group, and due to the spectral sensitivity difference between the color filters in which the spectral sensitivities are wide and narrow is obtained (steps S 1  and S 2 ). The level difference is corrected. The first captured color image and the second captured color image are combined with each other.

This application is a Continuation of PCT/JP2011/059431 filed on Apr.15, 2011, which claims priority under U.S.C. 119(a) to PatentApplication No. 2010-097367 filed in Japan on Apr. 20, 2010, all ofwhich are hereby expressly incorporated by reference into the presentapplication.

TECHNICAL FIELD

The present invention relates to an imaging apparatus equipped with asolid-state imaging device which can capture two kinds of color imageshaving different color tones, a method of processing a captured image,and a program for processing a captured image.

BACKGROUND ART

As an arrangement of color filters mounted in a solid-state imagingdevice, various arrangements have been proposed and used. For example,an imaging apparatus disclosed in Patent Document 1 below uses a colorfilter arrangement which is called the Bayer arrangement. In the Bayerarrangement, color filters of any one of the three primary colors of R(red), G (green), and B (blue) are arranged in a mosaic pattern inrespective pixels on the basis of a predetermined rule. In a red pixelin which a red filter is mounted, for example, green and blue signalstherefore cannot be detected. Consequently, an interpolation calculationis performed on signals of pixels which are in the periphery of the redpixel, and in which green and blue filters are respectively mounted,thereby obtaining the green and blue signals at the position of the redpixel.

In an imaging device disclosed in Patent Document 2 below, each twopixels which are adjacent to each other in an oblique direction arepaired, and color filters of one of the three primary colors of RGB arearranged in a mosaic pattern in the unit of paired pixels in accordancewith a predetermined rule. For example, a G1 color filter and a G2 colorfilter are mounted on a pair of pixels on each of which a green filteris mounted, respectively.

The relationship between G1 and G2 is set so that the color of G isattained by, for example, adding them to each other. The color filter ofG is produced so that the wavelength of 540 nm is set as the centerwavelength, and bell-shaped spectral characteristics having a width ofabout 100 nm in each of the back and front sides are obtained. Bycontrast, for example, G1 and G2 are separated in two filters so thatthe G1 filter detects a color having a wavelength of 440 nm to 540 nm,and the G2 filter detects a wavelength of 540 nm to 640 nm. With respectto the colors of R and B, similarly, R1 and R2 filters are mounted onpaired pixels, and B1 and B2 filters are mounted on paired pixels.

When, as described above, colors to be separated by color filters areseparated more finely than the case of the three colors of R, G, and B,the color reproducibility of an object image can be improved. However, acaptured image taken out from pixels on which the R1, G1, and B1 colorfilters are mounted, and that taken out from pixels on which the R2, G2,and B2 color filters are mounted are different in color tone from eachother. When the images are singly viewed, the images are color imageshaving an unnatural color tone. Therefore, adequate image combinationprocessing must be performed so that an object image having high colorreproducibility is obtained by image processing.

In an imaging apparatus disclosed in Patent Document 3 below, each pixelis divided into a small-area portion and a large-area portion. In eachof pixels on which, for example, a green (G) filter is mounted,moreover, the thickness of a filter mounted on the small-area portion ismade larger than that on the large-area portion, or that of an n-regionconstituting a photodiode is reduced.

As a result, the small-area portion cannot substantially detect incidentlight of a certain wavelength region, and the large-area portion candetect light of the wavelength region. By using this, the imagingapparatus detects whether light of the wavelength region exists or not,and determines the kind of the light source.

CITATION LIST Patent Literature

-   Patent Document 1: JP-A-2006-135468-   Patent Document 2: JP-A-2009-268078-   Patent Document 3: JP-A-2004-289728

SUMMARY OF INVENTION Technical Problem

As a conventional monitor apparatus which can display a color image, acathode ray tube type monitor apparatus (CRT) is usually used. Recently,however, a liquid crystal display apparatus is widely used as a liquidcrystal television receiver. As a result, an ordinary user is accustomedto view, for example, a vivid color image which is different in colortone from a natural color image. Therefore, situations where a colorimage captured by a digital camera seems not enough are increasing.

The above-described color filters which are used in conventionalsolid-state imaging devices are invented simply so that a colorreproduction of an object image is enabled to have a natural color tone.Therefore, a color image of an object cannot be captured as a vividcolor image.

On the other hand, there is also an imaging scene where, a color imageof an object must be captured as a vivid color image. A request for animaging apparatus which can capture both a vivid color image and anatural color image is strong.

It is an object of the invention to provide an imaging apparatusequipped with a solid-state imaging device which can capture two kindsof color images having different color tones, a method of processing acaptured image, and a program for processing a captured image.

Solution to Problem

An imaging apparatus of the invention, comprises: a solid-state imagingdevice including: a plurality of pixels which are arranged and formed ina two-dimensional array in a semiconductor substrate; a plurality ofcolor filters of a first color which are arranged and stacked inaccordance with a predetermined rule on a first pixel group thatincludes one of odd rows and even rows of the pixels; and a plurality ofcolor filters of a second color which are arranged and stacked inaccordance with a predetermined rule on a second pixel group thatincludes the other one of the odd rows and the even rows, the colorfilters of the second color being different in spectral sensitivity fromthe plurality of color filters of the first color; and an image processsection which obtains a level difference between captured image signalsof the pixels of the first pixel group and captured image signals of thepixels of the second pixel group, the level difference being due to aspectral sensitivity difference between the plurality of color filtersincluding the first color and the plurality of color filters includingthe second color, and which combines a first captured image that isobtained from the first pixel group by correcting the level difference,with a second captured image that is obtained from the second pixelgroup.

A method of processing a captured image which processes an imagecaptured by a solid-state imaging device of the invention comprises: aplurality of pixels which are arranged and formed in a two-dimensionalarray in a semiconductor substrate; a plurality of color filters of afirst color which are arranged and stacked in accordance with apredetermined rule on a first pixel group that includes one of odd rowsand even rows of the pixels; and a plurality of color filters of asecond color which are arranged and stacked in accordance with apredetermined rule on a second pixel group that includes the other oneof the odd rows and the even rows, the color filters of the second colorbeing different in spectral sensitivity from the plurality of colorfilters of the first color, in which, the method comprise: obtaining alevel difference between captured image signals of the pixels of thefirst pixel group and captured image signals of the pixels of the secondpixel group, the level difference being due to a spectral sensitivitydifference between the plurality of color filters including the firstcolor and the plurality of color filters including the second color;correcting the level difference; and combining a first captured imagethat is obtained from the first pixel group, and a second captured imagethat is obtained from the second pixel group with each other.

A program for processing a captured image which process an imagecaptured by the solid-state imaging device of the invention is providedwith a step of executing the method of processing a captured image.

Advantageous Effects of Invention

According to the invention, it is possible to capture two kinds of colorimages having different color tones (for example, a natural color imageand a vivid color image), and also to obtain an image which is acombination of the two color images.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a functional block diagram of an imaging apparatus of a firstembodiment of the invention.

FIG. 2 is a view of a color filter arrangement in a solid-state imagingdevice shown in FIG. 1.

FIG. 3 is a view showing spectral sensitivities of color filters used inthe color filter arrangement shown in FIG. 2.

FIG. 4 is a flowchart showing the procedure of captured image processingin the first embodiment of the invention.

FIG. 5 is a functional block diagram of an imaging apparatus of a secondembodiment of the invention.

FIG. 6 is a flowchart showing the procedure of captured image processingin the second embodiment of the invention.

FIG. 7 is a functional block diagram of an imaging apparatus of a thirdembodiment of the invention.

FIG. 8 is a view illustrating edge determination on an image.

FIG. 9 is a flowchart showing the procedure of captured image processingin the third embodiment of the invention.

FIG. 10 is a functional block diagram of an imaging apparatus of afourth embodiment of the invention.

FIG. 11 is a flowchart showing the procedure of captured imageprocessing in the fourth embodiment of the invention.

FIG. 12 is a view of a color filter arrangement in a solid-state imagingdevice an embodiment which is different from that of FIG. 2.

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of the invention will be described withreference to the drawings.

FIG. 1 is a functional block diagram of an imaging apparatus of a firstembodiment of the invention. The imaging apparatus 10 includes asolid-state imaging device 11, an imaging control section 13, a memory14, and a CPU 15. The imaging control section 13 drives and controls thesolid-state imaging device 11. Furthermore, the imaging control section13 receives a captured image signal output from the solid-state imagingdevice 11, performs a correlated double sampling process, a gain controlprocess, and an A/D (analogdigital) conversion process, and outputs theresulting signal to a bus 12. The memory 14 is connected to the bus 12.The CPU 15 generally controls the whole imaging apparatus 10. The CPU 15incorporates a DSP function, and performs a correction process (anoffset process, a gamma correction process, an RGB/YC conversionprocess, a concurrent process, and the like) on the captured imagesignal output from the solid-state imaging device 11, to produce anobject image.

The imaging apparatus 10 further includes a narrow-spectral pixel meanvalue calculation circuit 17, a wide-spectral pixel mean valuecalculation circuit 18, a color reproduction selection section 19, acorrection ratio calculation section 20, and a multiplier 21. Thenarrow-spectral pixel mean value calculation circuit 17 and thewide-spectral pixel mean value calculation circuit 18 process thedigital captured image signal received from the bus 12. The colorreproduction selection section 19 selectively controls the outputs ofthe calculation circuits 17, 18 and digital captured image signalsreceived from the bus 12, and performs color reproduction. Thecorrection ratio calculation section 20 calculates a correction ratiofrom the output signal of the calculation circuit 17 or 18 which isselected by the color reproduction selection section 19. The multiplier21 multiplies a signal of a pixel which is selected by the colorreproduction selection section 19 in accordance with the digitalcaptured image signals received from the bus 12, and in which the leveldifference is to be corrected, by an output signal of the correctionratio calculation section 20, and returns the product to the bus 12.

The calculation circuits 17, 18, the color reproduction selectionsection 19, the correction ratio calculation section 20, and themultiplier 21 constitute a part of an image process section. Itsoperation will be described in detail later.

FIG. 2 is a surface diagram of the solid-state imaging device 11. In thesolid-state imaging device 11 in the embodiment, a plurality of pixels(photodiodes) having the same light receiving area are arranged andformed in a two-dimensional array in a semiconductor substrate. The evenpixel rows are arranged to be shifted by ½ pixel pitch with respect tothe odd pixel rows. When only the arrangement of the pixels of the oddrows is considered, the pixels constitute a square lattice arrangement.When only the arrangement of the pixels of the even rows is considered,the pixels constitute a square lattice arrangement. Color filters R, G,B of the three primary colors are Bayer-arranged in the pixels (A-grouppixels) of the odd rows, and color filters r, g, b of the three primarycolors are Bayer-arranged in the pixels (B-group pixels) of the evenrows.

Although not illustrated, a signal readout section which reads outcaptured image signals detected by the pixels to the outside is formedin the surface portion of the semiconductor substrate. In the case wherethe solid-state imaging device is of the CCD type, as disclosed in, forexample, JP-A-2005-72966, the signal readout section may be configuredby vertical charge transfer paths, a horizontal charge transfer path,and an amplifier which outputs a voltage value signal corresponding tothe amount of signal charges as a captured image signal. In the casewhere the solid-state imaging device is of the CMOS type, as disclosedin, for example, JP-A-2007-124137 and JP-A-2007-81140, the signalreadout section may be configured by MOS transistors, a vertical scancircuit, a horizontal circuit, etc.

FIG. 3 is a graph showing spectral sensitivities of the color filtersRGB and rgb. The wavelength of light is represented by λ. The spectralsensitivity of each of the red color filters R(λ), green color filtersG(λ), and blue color filters B(λ) which are stacked on the A-grouppixels has a bell-like mountain shape and a large width (hereinafter,this situation is referred to as wide).

By contrast, although the spectral sensitivity of each of the red colorfilters r(λ), green color filters g(λ), and blue color filters b(λ)which are stacked on the B-group pixels has a bell-like mountain shape,it has a small width ((hereinafter, this situation is referred to asnarrow).

The graph of FIG. 3 shows the spectral sensitivities of light of variouswavelengths. As described with reference to FIG. 2, the A- and B-grouppixels have the same light receiving area. Therefore, the graph of FIG.3 shows the wavelength dependency of the amount of received light whichhas passed through the color filter of each pixel.

It is difficult that light on the long wavelength side (infrared region)of a red color filter is cut off only by a color filter. Usually,therefore, an infrared cut filter is inserted in an imaging lens systemof a digital camera. The graph of FIG. 3 shows characteristics in thecase where only red color filters R or r in which an infrared cut filteris not inserted are used.

In an upper end portion of FIG. 3, relationships between the value ofthe wavelength λ (nm) and a color. The wide red filters R which are tobe stacked on the A-group pixels are produced actually as filters thatallow orange light to pass therethrough, and also part of yellow lightto pass therethrough. The green filters G are produced as filters that,on the long wavelength side of them, allow part of yellow and orangelight to pass therethrough. The same applies to the short wavelengthside. Wide green filters G are produced as filters that allow part ofblue light to pass therethrough, and wide blue filters B are produced asfilters that allow part of green light to pass therethrough.

Namely, both the pixels on which the G filters are mounted, and those onwhich the R filters are mounted receive orange and yellow light, andboth the pixels on which the G filters are mounted, and those on whichthe B filters are mounted receive part of light in vicinity of theboundary of blue and green.

In the case where an object color image is reproduced based on detectionsignals of the pixels on which the G filters are mounted, those of thepixels on which the R filters are mounted, and those of the pixels onwhich the B filters are mounted, the larger the overlapping portions ofthe detection signals (in the above, the boundary portion of orange andyellow, and that of blue and green), the larger rate at which green andred, and green and blue are mixed in the reproduced color image.

Namely, color signals R, G, B which are obtained through wide filters R,G, B are color signals which contain not only the respective originalcolors of red, green, and blue but also other colors, and in which themixture fraction (mixture ratio) of the original color is low. At amixture ratio where the degree of mixture with other colors is adequate,however, a natural color image is obtained.

By contrast, the smaller the overlapping portions, the higher themixture fraction (mixture ratio) of the original color the signals have.Namely, color signals r, g, b which are obtained through narrow filtersr, g, b are color signals in which the mixture fraction (mixture ratio)of the original color is high. A color image which is reproduced basedon the color signals r, g, b is a vivid color image.

Colors in a certain wavelength range are not identical, but aregradually changed. In the wavelength range of 500 nm to 570 nm of FIG.3, for example, all colors are illustrated as “green”. But, actually,colors in which the mixture ratio of blue is gradually increased asfurther advancing toward the left side of FIG. 3 are obtained, and thosein which the mixture ratio of yellow is gradually increased as furtheradvancing toward the right side of FIG. 3 are obtained. In the above,“original color” in the case of, for example, green does not mean colorsin the whole range of 500 nm to 570 nm, but means colors in a part ofthe range, for example, a predetermined wavelength range (for example,±25 nm) which is centered at 535 nm.

In the solid-state imaging device 11 which is to be mounted in theimaging apparatus of the embodiment, the wide color filters R, G, B arestacked on the A-group pixels, so that a natural color image of anobject can be captured, and the narrow color filters r, g, b are stackedon the B-group pixels, so that a vivid color image of the object can becaptured.

As the wide color filters R, G, B, color filter materials whichconventionally exist, and which allow a natural color image to becaptured are used. In contrast to the wide color filter materials, thefilters r, g, b having the spectral characteristics exemplarily shown inFIG. 3 are produced by pigments or dyes which are different frommaterials for a wide color filter.

In this case, when the thickness of the wide color filters R, G, B isdifferent from that of the narrow color filters r, g, b, unevenness isformed on the surface of the solid-state imaging device 11, and steps ofstacking a microlens thereon are complicated. In order to facilitate theproduction to reduce the production cost, therefore, the wide colorfilters R, G, B and the narrow color filters r, g, b preferably have thesame thickness.

Preferably, the peak sensitivity values of the filters R, G, B are setto be approximately equal respectively to those of the filters r, g, bso that the brightness of a natural color image captured by the A-grouppixels is substantially equal to that of a vivid color image captured bythe B-group pixels.

In the example of FIG. 3, in the case where the error between the peaksensitivity value Bw of the blue filter G and the peak sensitivity valuebn of the blue filter b is within 10%, it is deemed that the peaksensitivity value of the blue filter G is equal to the sensitivity valueof the blue filter b. In the case where the error between the peaksensitivity value Gw of the green filter G and the peak sensitivityvalue bn of the green filter g is within 10%, it is deemed that the peaksensitivity value of the green filter G is equal to that of the greenfilter g. In the case where the error between the peak sensitivity valueRw of the red filter R and the peak sensitivity value m of the redfilter r is within 10%, furthermore, it is deemed that the peaksensitivity value of the red filter R is equal to that of the red filterr.

Preferably, the wavelength λ_(BW) of light providing the peaksensitivity value Bw of blue light is identical with the wavelengthλ_(bn) of light providing the peak sensitivity value bn. Actually,however, the materials forming the filter B are different from thoseforming the filter b, and therefore it is difficult to make thewavelength λ_(BW) identical with the wavelength λ_(bn). However, theerror between the wavelength λ_(BW) and the wavelength λ_(bn) may bewithin the range of about ±20 nm. This is similarly applicable also tothe other colors.

The wavelength range of the spectral sensitivity of the narrow redfilter r is completely inside that of the spectral sensitivity of thewide red filter R. This is similarly applicable also to the other colors(green and blue). In the wavelength range (about 380 nm to 650 nm) ofvisible light, namely, R(λ)>r(λ), G(λ)>g(λ), and B(λ)>b(λ) are set

This is because a color image captured through the wide filters R, G, Bis a natural color image, but a color image captured at spectralsensitivities which are deviated from the wavelength ranges of thespectral sensitivities of the filters R, G, B is a color image having anunnatural color tone. When the wavelength range of the spectralsensitivity of the narrow filter r (g, b) is completely inside that ofthe spectral sensitivity of the wide filter R (G, B), a color imagecaptured through the narrow filters r, g, b is a vivid color image butnot an image having an unnatural color tone.

In the case where the wavelength range of the spectral sensitivity ofthe narrow green filter g is set to be inside that of the spectralsensitivity of the wide green filter G, it is preferable that thespectral sensitivity of the narrow green filter g is separated by anequal distance from the right and left edge lines of the mountain shapeof the spectral sensitivity of the wide green filter G. However, thematerials of the filter g are different from those of the filter G.Therefore, the above is not essential, and the spectral sensitivity ofthe narrow green filter g may be slightly shifted to one side.

As described above, the mixture ratio (content ratio) of the originalcolor contained in transmitted light from the narrow color filters r, g,b is higher than the mixture ratio (content ratio) of the original colorcontained in transmitted light from the wide color filters R, G, B.Next, this will be quantitatively described.

When the mixture ratio of the original color in the wide color filtersR, G, B is indicated by α, and the mixture ratio of the original colorin the narrow color filters r, g, b is indicated by α+Δα, it ispreferable to set Δα>0. According to this, it is possible that an imagecaptured through the narrow color filters r, g, b can be viewed as beingvisually vivid.

It is subjectively determined whether an image is vivid or not.Therefore, the degree of the difference Δα is hardly quantified. In theembodiment, the mixture ratio α is defined in the following manner.Referring to FIG. 3, the half-value width Bd with respect to the peaksensitivity value Bw of the blue color filter B is obtained, and thehalf-value width bd with respect to the peak sensitivity value bn of theblue color filter b is obtained. Then, the mixture ratio (content ratio)of the original color contained in transmitted light from the wide colorfilter B is defined as Bd/Bw, and the mixture ratio (content ratio) ofthe original color contained in transmitted light from the narrow colorfilter b is defined as bd/bn. At this time, the spectral sensitivity ofthe narrow color filter b is determined so as to attain:bd/bn<Bd/Bw.

When Δα above is used in the expression, the spectral sensitivity of thenarrow color filter b is determined so as to attain:bd/bn+Δα=Bd/Bw.

In the measurement values of the example of FIG. 3, bd/bn=53.75(%) andBd/Bw=64.7(%). Therefore, Δα=about 10(%).

With respect to green, similarly, the spectral sensitivity of the narrowcolor filter g is determined so as to attain:gd/gn<Gd/Bw.

When Δα above is used in the expression, the spectral sensitivity of thenarrow color filter g is determined so as to attain:gd/gn+Δα=Gd/Gw.

In the measurement values of the example of FIG. 3, gd/gn=63.75(%) andGd/Gw=95.12(%). Therefore, Δα=about 31(%).

With respect to red, as described above, the long wavelength side is cutoff by the infrared cut filter. Therefore, the determination isperformed by using only the short wavelength side of the position of thepeak sensitivity.

The spectral sensitivity of the narrow color filter r is determined soas to attain:rd/rn<Rd/Rw.

When Δα above is used in the expression, the spectral sensitivity of thenarrow color filter r is determined so as to attain:rd/rn+Δα=Rd/Rw.

In the measurement values of the example of FIG. 3, rd/rn=7.5(%) andRd/Rw=34(%). Therefore, Δα=about 26(%).

When, as described above, the content ratio of the original colorcontained in transmitted light from the narrow color filters r, g, b ismade higher than that of the original color contained in transmittedlight from the wide color filters R, G, B, the single solid-stateimaging device 11 can simultaneously capture two object images (anatural color image obtained from the A-group pixels, and a vivid colorimage obtained from the B-group pixels). In the embodiment, a colorwithin the wavelength range of the half-value width of each of thespectral sensitivities of the narrow color filters r, g, b is set as“original color”, so that the content ratios of the original color aremade higher than those of the wide color filters R, G, B.

As described above, the solid-state imaging device 11 having theconfiguration of FIG. 2 can simultaneously capture two color images.However, each of the captured images is a captured image which uses ½ ofthe number of the pixels mounted in the solid-state imaging device 11,and there is also a request to capture a high-definition object imagewhich uses all the pixels. However, the A-group pixels and the B-grouppixels have the color filters of different spectral sensitivities, andtherefore a level difference (sensitivity ratio) is produced between thecaptured images. Even when a high-definition image is obtained by simplycombining the captured images with each other, the high-definition imageis an image giving a feeling of strangeness.

In the following embodiments, therefore, a method of processing acaptured image will be described in which a high-definition image thatdoes not give a feeling of strangeness is produced from signals of anobject color image having a natural color tone, and those of an objectcolor image having a vivid color tone.

When the A-group pixels and the B-group pixels are different in spectralsensitivity from each other, different pixel values are obtained withrespect to the same color contained in the subject to be captured(object), and a level difference is produced. As seen from FIG. 3,moreover, the level difference between the A-group pixels and theB-group pixels is different depending on respective colors (wavelengths)in the visible light region. Therefore, a highly accurate correctioncannot be performed by a process in which the whole screen is uniformlyprocessed, such as the gain process, and local optimization is required.

In the following embodiments, therefore, the level difference iscorrected in the following manner. The mean value calculation circuits17, 18 in FIG. 1 calculate mean values (for example, a mean value ofneighboring 5×5 pixels of the same color as a to-be-corrected pixel) ofsame-color pixels (R and r, G and g, and B and b) for each spectralsensitivity. The correction ratio calculation section 20 calculates aratio of the mean value of the spectral (assumed to be the wide side)selected by the color reproduction selection section 19, and the othermean value (the narrow side). Then, the multiplier 21 multiples thedetection signal of a pixel which is not selected, and in which thelevel difference is to be corrected, by the ratio calculated by thecorrection ratio calculation section 20.

The color reproduction selection section 19 is used for selectingwhether the color reproduction is set to a vivid color on the narrowside or a natural color on the wide side. In the case of landscapeimaging, for example, the narrow side is selected, and, in the case ofperson imaging, the wide side is selected.

When two kinds of color filters having different spectral sensitivitiesor narrow and wide are used as described above, it is possible to selecteither of two kinds of color reproductions. When the level of thespectral sensitivity of the unselected color reproduction is correctedto match with that of the selected other color reproduction, moreover,RGB (rgb) can be interpolated by using both the A-group pixels and theB-group pixels, and a high-resolution object image can be produced.

FIG. 4 is a flowchart showing the procedure of a processing program fora method of processing a captured image of the first embodiment of theinvention. When the user selects a vivid color image, or when an imagingscene is set to the landscape imaging, captured image signals of theB-group pixels on which the narrow filters r, g, b are stacked can beused as they are, and therefore the to-be-corrected pixels are set tothe A-group pixels (pixels on which the wide filters R, G, B arestacked).

First, the mean value of captured image signals of a predeterminednumber of pixels in the periphery of a to-be-corrected pixel, forexample, 5×5 same-color pixels which are in the vicinity of theto-be-corrected pixel, and which are on the narrow side is calculated(step S1). Next, the mean value of captured image signals of 5×5same-color pixels which are in the periphery of the to-be-correctedpixel, and which are on the wide side is calculated (step S2).

In step S3, it is determined whether the color reproduction is on thenarrow side or on the wide side. In the example, the color reproductionis performed on the narrow side as described above, and therefore theprocess advances from step S3 to step S4 to set [mean value obtained instep S1]/[mean value obtained in step S2]=correction ratio.

In next step S5, the value of the captured image signal of theto-be-corrected pixel (in the example, the pixel on the wide side) ismultiplied by the correction ratio calculated in step S4, and the leveldifference (sensitivity ratio) is corrected so as to have the same levelas that of the narrow pixel.

In step S6, it is determined whether the correction is ended withrespect to all of to-be-corrected pixels or not. If the correction isnot ended, the process proceeds to step S7 to change the to-be-correctedpixel to the next to-be-corrected pixel, and returns to step S1.

When the user selects a natural color image, or when an imaging scene isset to the person imaging, captured image signals of the A-group pixelson which the wide filters R, G, B are stacked can be used as they are,and therefore the to-be-corrected pixels are set to the B-group pixels(pixels on which the narrow filters r, g, b are stacked).

In this case, the process proceeds to step S1, step S2, and step S3, thewide side is selected in the determination of step S3, and the processfurther proceeds to step S8. In step S8, correction ratio=[mean valueobtained in step S2]/[mean value obtained in step S1] is calculated. Innext step S9, the value of the captured image signal of theto-be-corrected pixel (in the example, the pixel on the narrow side) ismultiplied by the correction ratio calculated in step S8, and the leveldifference is corrected so as to have the same level as that of the widepixel.

After step S9, the process proceeds to step S6 to determine whether thecorrection process is ended with respect to all of to-be-correctedpixels or not. If the correction process with respect to all ofto-be-corrected pixels is ended, the process is terminated.

After the correction process of FIG. 4 is ended, a process of combiningthe captured image due to the A-group pixels with that due to theB-group pixels (of course, the captured image due to one of the pixelgroups is a corrected captured image) is executed. This is similarlyapplicable also to the following embodiments.

FIG. 5 is a functional block diagram of an imaging apparatus 30 of asecond embodiment of the invention. The second embodiment is differentfrom the first embodiment shown in FIG. 1 in that a correction ratiosuppression section 23 is disposed in rear of the correction ratiocalculation section 20 and in front of the multiplier 21, and identicalwith the first embodiment in the other points. The correction ratiosuppression section 23 previously sets an upper limit value (forexample, the spectral sensitivity ratio of the same color) with respectto a correction ratio, and, when the correction ratio calculated by thecorrection ratio calculation section 20 exceeds the upper limit value(upper limit correction ratio), performs a suppression process in whichthe correction ratio is replaced with the upper limit correction ratio,and the resulting correction ratio is output to the multiplier 21.

FIG. 6 is a flowchart showing the procedure of captured image processingwhich is executed by the imaging apparatus 30 of the second embodiment.Steps which are identical with those of the flowchart of FIG. 4 aredenoted by the same step numbers, and their detailed description isomitted. Only different points will be described.

In the embodiment, as compared with the flowchart of FIG. 4, steps S11and S12 are disposed between step S4 and step S5, and steps S11 and S12are disposed also between step S8 and step S9.

In step S11, it is determined whether the correction ratio calculated instep S4 or step S8 is smaller than the upper limit correction ratio ornot. If the calculated correction ratio is smaller than the upper limitcorrection ratio (the determination result is Yes), the process proceedsto step S5 or step S9 to perform correction by using the calculatedcorrection ratio. If the result of the determination of step S11 isnegative (No), i.e., if the correction ratio calculated in step S4 orstep S8 exceeds the upper limit correction ratio, the process proceedsto step S12 to replace the correction ratio with the upper limitcorrection ratio, and then to step S5 or step S9.

According to the embodiment, the correction ratio is suppressed, andtherefore it is possible to suppress resolution reduction caused byovercorrection in the case of capturing a scene containinghigh-frequency components.

FIG. 7 is a functional block diagram of an imaging apparatus 40 of athird embodiment of the invention. The imaging apparatus 40 of theembodiment is different from the imaging apparatus 10 of the firstembodiment shown in FIG. 1 in that an edge determination section 25 isdisposed in front of the mean value calculation circuit 17, and an edgedetermination section 26 is disposed in front of the mean valuecalculation circuit 18, and identical with the first embodiment in theother configuration. The edge determination section 25 determineswhether an edge exists in the captured image due to the narrow pixels(B-group pixels) or not. The edge determination section 26 determineswhether an edge exists in the captured image due to the wide pixels(A-group pixels) or not.

The edge determination section 25 determines the existence ornonexistence of an edge based on the level difference between capturedimage signals of the B-group pixels on which the narrow filters r, g, bare stacked. The edge determination section 26 determines the existenceor nonexistence of an edge based on the level difference betweencaptured image signals of the A-group pixels on which the wide filtersR, G, b are stacked.

As described above, the mean value calculation circuits 17, 18 use themean value of same-color pixels in the periphery of the to-be-correctedpixel. When an edge portion is included in the same-color pixels in theperiphery of the to-be-corrected pixel, however, a pixel value in whichthe level difference is large with respect to other pixel values entersthe mean value, thereby causing a possibility that the correction iserroneously performed. Therefore, the edge determination sections 25, 26determine the existence or nonexistence of an edge, and the mean valuecalculation circuits 17, 18 calculate mean values of pixels in which, asshown in FIG. 8, edge portions that are determined to include an edgeare excluded. More specifically, FIG. 8 shows an example in which it isdetermined that an edge is included in the lower right portion which isnot enclosed by the thick line. While excluding the lower right portionwhich is not enclosed by the thick line, the mean value calculationcircuits 17, 18 calculate mean values by using captured image signals ofthe remaining pixels (the pixels in the region enclosed by the thickline).

FIG. 9 is a flowchart showing the procedure of captured image processingin the third embodiment. The flowchart of FIG. 9 is different from thatof FIG. 4 in that steps S21, S22, and S23 are disposed in front of stepS1, and that steps S25, S26, and S27 are disposed between step S1 andstep S2, and identical with the flowchart of FIG. 4 in the other points.Hereinafter, therefore, only different points will be described.

In step S21, it is determined whether the level difference between thecaptured image signal of the to-be-corrected pixel and captured imagesignals of pixels (narrow pixels) in the periphery of the determinationobject is smaller than a threshold or not. If the level difference issmaller than the threshold (Yes in step S21), the pixels in theperiphery of the determination object are flagged as pixels which are tobe used in calculation of a mean value (step S22), and the process thenproceeds to step S23.

If, in the result of the determination of step S21, the level differenceis equal to or larger than the threshold (No in step S21), the processsimilarly proceeds to step S23. Then, it is determined whether thedetermination of step S21 is performed on all of the periphery pixels ornot (step S23). If the determination of step S21 is not ended withrespect to all of the periphery pixels (No in step S23), the peripherypixel of the determination object is changed, and the process returns tostep S21. By contrast, if the determination of step S21 is ended withrespect to all of the periphery pixels (Yes in step S23), the processproceeds to step S1. In the mean value calculation process of step S1,the mean value of the periphery pixels in which a flag is set in stepS22 is calculated.

Basic processes of S25, S26, and S27 are identical with those of stepsS21, S22, and S23, respectively, but different in that the peripherypixels are wide pixels.

According to the embodiment, an edge portion in an image is determined,and the mean value is calculated while excluding the edge portion.Therefore, erroneous correction caused by an edge portion in an imagecan be reduced.

FIG. 10 is a functional block diagram of an imaging apparatus 50 of afourth embodiment of the invention. The imaging apparatus 50 of theembodiment is an imaging apparatus which can combine a natural colorimage captured by the A-group pixels on which the wide filters R, G, Bare mounted, with a vivid color image captured by the B-group pixels onwhich the narrow filters r, g, b are mounted, thereby producing oneimage having a wide dynamic range.

In the case where an image having a wide dynamic range is to beproduced, addition of two or narrow and wide pixels which are of thesame color, and which are adjacent to each other is performed. When thelevel difference between narrow and wide pixels due to the spectralsensitivity ratio is not adequately corrected, however, an unnaturalimage is produced by the combination.

Similarly with the imaging apparatus 10 of FIG. 1, the imaging apparatus50 of the embodiment includes the solid-state imaging device 11, the bus12, the imaging control section 13, the memory 14, the CPU 15, thecorrection ratio calculation section 20, and the multiplier 21. Theimaging apparatus 50 further includes a D (dynamic) range setting device31 which controls the imaging control section 13, and an exposuredifference correction amount calculating device 32 which is controlledby the D-range setting device 31.

The imaging apparatus 50 further includes: a first color reproductionselection section (I) 33; a multiplier 34; an adequate-exposure pixelmean value calculation circuit 35; an underexposure pixel mean valuecalculation circuit 36; and a second color reproduction selectionsection (II) 37. The first color reproduction selection section (I) 33receives the captured image signal of the narrow pixels, and that of thewide pixels, and selects one of the signals. The multiplier 34multiplies one of the captured image signal of the narrow pixels, andthat of the wide pixels which is selected by the color reproductionselection section 33, by an output signal of the exposure differencecorrection amount calculating device 32. The adequate-exposure pixelmean value calculation circuit 35 calculates a mean value of an outputsignal the multiplier 34, as a mean value of adequate-exposure pixels.The underexposure pixel mean value calculation circuit 36 calculates amean value of the other of the captured image signal of the narrowpixels, and that of the wide pixels, as a mean value of underexposurepixels. The second color reproduction selection section (II) 37 selectsone of the captured image signals of the narrow and wide pixels whichare received from the bus 12, and outputs the selected signal to themultiplier 21. The outputs of the mean value calculation circuits 35, 36are received by the correction ratio calculation section 20, and thecorrection ratio is calculated, and then output to the multiplier 21.

FIG. 11 is a flowchart showing the process procedure of a captured imageprocessing program which is executed in the imaging apparatus 50. First,it is determined whether a D-range which is set by the user, or which isautomatically set by the imaging apparatus 50 is 100%, 200%, or 400%(step S31).

If the D-range is 100%, the process proceeds to step S32 to set exposuredifference correction amount=1 time. If the D-range is 200%, the processproceeds to step S33 to set exposure difference correction amount=½times. If the D-range is 400%, the process proceeds to step S34 to setexposure difference correction amount=¼ times.

The exposure difference correction amount is the difference of exposuretimes. When the exposure time of the B-group pixels is set to beidentical with that of the A-group pixels, the D-range is 100%. When theexposure time of one of the pixel groups is set to be ½ of that of theother pixel group, the D-range is 200%. When the exposure time of one ofthe pixel groups is set to be ¼ of that of the other pixel group, theD-range is 400%.

The determination of which one of the pixel groups is made shorter inexposure time is performed depending whether the color reproduction ison the wide side (natural color tone) or on the narrow side (vivid colortone). Hereinafter, pixels belonging to a pixel group, i.e., the A-grouppixels or the B-group pixels in which the exposure time is set to beshorter than that in the other pixel group are referred to asunderexposure pixels, and imaging which is performed while the exposuretime is set to be short is referred to as underexposure imaging.Hereinafter, pixels belonging to a pixel group in which the exposuretime is not set to be short are referred to as adequate-exposure pixels,and imaging under such conditions is referred to as adequate-exposureimaging.

After steps S32, S33, and S34, the process proceeds to step S35 todetermine whether the color reproduction is on the narrow side (the sidewhere a vivid color tone is obtained) or on the wide side (the sidewhere a natural color tone is obtained). In the case where the colorreproduction is set to the narrow side (the side where the color tone isvivid), the process proceeds to step S36, the narrow pixels (B-grouppixels) which are on the side of the color reproduction are subjected tounderexposure imaging, and the wide pixels (A-group pixels) aresubjected to adequate imaging. These imaging processes aresimultaneously performed, and underexposure imaging is conducted duringthe exposure time when adequate imaging is conducted.

In step S37, similarly in steps S1 and S2 in FIG. 4, the mean value ofcaptured image signals of a predetermined number of narrow pixels of thesame color in the periphery of a to-be-corrected pixel is obtained, andthat of captured image signals of a predetermined number of wide pixelsof the same color in the periphery of the to-be-corrected pixel isobtained. Then, the correction ratio is obtained as:

correction ratio=[mean value of narrow pixels]/[mean value of widepixels]×exposure difference correction amount.

In step S38, the captured image signal of the to-be-corrected pixel (inthis case, a wide pixel) is multiplied by the correction ratio, thecaptured image signal of the wide pixel is matched with the spectralsensitivity of a narrow pixel, also the exposure difference (thedifference in exposure time) is matched, and then the process is ended.Of course, the above process is performed on all to-be-corrected pixels,but the illustration of steps S6 and S7 in FIG. 4 is omitted in theflowchart of FIG. 11. Steps S6 and S7 may be sequentially disposed afterstep S38, and the return destination of step S7 is set to step S37.

In the case where, in step S35, the color reproduction is set to thewide side (the side where the color tone is natural), the processproceeds from step S35 to step S36. In this case, the wide pixels(A-group pixels) which are on the side of the color reproduction aresubjected to underexposure imaging, and the narrow pixels (B-grouppixels) are subjected to adequate imaging.

In step S40, the mean value of captured image signals of a predeterminednumber of narrow pixels of the same color in the periphery of theto-be-corrected pixel is obtained, and that of captured image signals ofa predetermined number of wide pixels of the same color in the peripheryof the to-be-corrected pixel is obtained. Then, the correction ratio isobtained as:

correction ratio=[mean value of wide pixels]/[mean value of narrowpixels]×exposure difference correction amount.

In step S41, the captured image signal of the to-be-corrected pixel (inthis case, a narrow pixel) is multiplied by the correction ratio, thecaptured image signal of the narrow pixel is matched with the spectralsensitivity of a wide pixel, also the exposure difference (thedifference in exposure time) is matched, and then the process is ended.Of course, the above process is performed on all to-be-corrected pixels.

In step S36 and step S39 in FIG. 11, Under imaging and Adequate imagingare written. In the case where the D-range is 100%, however, theexposure time of a narrow pixel is equal to that of a wide pixel.Therefore, imaging is performed without distinguishing between “under”and “adequate”.

The above-described program for processing a captured image can beexecuted not only in the case where it is incorporated in an imagingapparatus, but also in an external personal computer. It can be used inthe case where one high-definition color image is produced by combiningtwo color images captured by the imaging apparatus of the embodiment,and in the case where a color image having a wide dynamic range isproduced by combination. The program for processing a captured image maybe stored in a recording medium such as a hard disk or a ROM, and, whenit is to be executed by a CPU or a processor, read out into a RAM or thelike. Alternatively, the program for processing a captured image may bestored on a recording medium such as a CD-ROM.

The solid-state imaging device of the embodiment has been described as asolid-state imaging device in which pixels are arranged in a checkeredpattern as shown in FIG. 2. The pixel arrangement is not limited to theembodiment. As shown in FIG. 12, for example, all pixels may be arrangedin a square lattice (illustrated as a color filter arrangement). In thesolid-state imaging device 22 of FIG. 12, color filters of the threeprimary colors are Bayer-arranged in the pixels (A-group pixels) of theodd rows, and color filters of the three primary colors areBayer-arranged also in the pixels (B-group pixels) of the even rows.

Also in the solid-state imaging device 22 of the embodiment, wide colorfilters R, G, B are stacked on the A-group pixels, narrow color filtersr, g, b are stacked on the B-group pixels, and a natural color image ofan object, and a vivid color image of the object can be simultaneouslycaptured by one imaging operation.

In the above-described embodiments, the solid-state imaging device inwhich the color filters that allow a natural color image to be captured,and those that allow a vivid color image to be captured are mounted hasbeen exemplarily described. However, the image processing methods of theabove-described embodiments are not limited to such filters, and may beapplied to captured images of a solid-state imaging device which cancapture two kinds of color images having different color tones.

In the above-described embodiments of image processing, the colorfilters of the three primary colors have been exemplarily described.However, the methods can be similarly applied also to complementarycolor filters (color filters in which a respective one of the threeprimary colors is omitted). The above-described image processing methodscan be applied even when the narrow spectral sensitivities are notcompletely inside the wide spectral sensitivities.

The narrow/wide relationships with respect to complementary colorfilters are as follows. The complementary color of red (R) is cyan(B+G), that of blue (B) is yellow (G+R), and that of green (G) ismagenta (B+R). Here, the narrow/wide relationship of cyan is as follows.Namely, B which constitutes wide cyan, and B which constitutes narrowcyan have a narrow/wide relationship, and also G which constitutes widecyan, and G which constitutes narrow cyan have a narrow/widerelationship.

The same is applicable to the other colors or yellow and magenta. Thewavelength range of the half-value widths with respect to the spectralsensitivities of G and R constituting narrow yellow is inside that ofthe half-value widths with respect to the spectral sensitivities of Gand R constituting wide yellow. Furthermore, the wavelength range of thehalf-value widths with respect to the spectral sensitivities of B and Rconstituting narrow magenta is inside that of the half-value widths withrespect to the spectral sensitivities of B and R constituting widemagenta.

Moreover, the description has been made on the assumption that thepixels have the same area. When the areas of pixels in each of thegroups are equal to one another within an error range, it is notnecessary that the areas of the A-group pixels and those of the B-grouppixels are equal to each other within an error range. Theabove-described image processing methods can be applied even when theA-group pixels and the B-group pixels are not equal in area to eachother.

A described imaging apparatus of the embodiments comprises: asolid-state imaging device including: a plurality of pixels which arearranged and formed in a two-dimensional array in a semiconductorsubstrate; a plurality of color filters of a first color which arearranged and stacked in accordance with a predetermined rule on a firstpixel group that includes one of odd rows and even rows of the pixels;and a plurality of color filters of a second color which are arrangedand stacked in accordance with a predetermined rule on a second pixelgroup that includes the other one of the odd rows and the even rows, thecolor filters of the second color being different in spectralsensitivity from the plurality of color filters of the first color; andan image process section which obtains a level difference betweencaptured image signals of the pixels of the first pixel group andcaptured image signals of the pixels of the second pixel group, thelevel difference being due to a spectral sensitivity difference betweenthe plurality of color filters including the first color and theplurality of color filters including the second color, and whichcombines a first captured image that is obtained from the first pixelgroup by correcting the level difference, with a second captured imagethat is obtained from the second pixel group.

Also, a method of processing a captured image which processes an imagecaptured by a solid-state imaging device of the embodiments comprises: aplurality of pixels which are arranged and formed in a two-dimensionalarray in a semiconductor substrate; a plurality of color filters of afirst color which are arranged and stacked in accordance with apredetermined rule on a first pixel group that includes one of odd rowsand even rows of the pixels; and a plurality of color filters of asecond color which are arranged and stacked in accordance with apredetermined rule on a second pixel group that includes the other oneof the odd rows and the even rows, the color filters of the second colorbeing different in spectral sensitivity from the plurality of colorfilters of the first color, in which the method comprise: obtaining alevel difference between captured image signals of the pixels of thefirst pixel group and captured image signals of the pixels of the secondpixel group, the level difference being due to a spectral sensitivitydifference between the plurality of color filters including the firstcolor and the plurality of color filters including the second color;correcting the level difference; and combining a first captured imagethat is obtained from the first pixel group, and a second captured imagethat is obtained from the second pixel group with each other.

Also, in the imaging apparatus and the method of processing a capturedimage of the embodiments, the image process section obtains the leveldifference from a first mean value which is a mean value of the capturedimage signals of a predetermined number of pixels that are in aperiphery of a pixel that is a correction subject, and that belong tothe first pixel group, and a second mean value which is a mean value ofthe captured image signals of a predetermined number of pixels that arein a periphery of the pixel that is the correction subject, and thatbelong to the second pixel group, and corrects the level difference.

Also, in the imaging apparatus and the method of processing a capturedimage of the embodiments, the image process section corrects the leveldifference by: setting a ratio of the first mean value and the secondmean value, as a correction ratio; and multiplying the captured imagesignal of the pixel that is the correction subject, by the correctionratio.

Also, in the imaging apparatus and the method of processing a capturedimage of the embodiments, when the correction ratio exceeds a presetupper limit value, the image process section uses the upper limit valueas the correction ratio.

Also, in the imaging apparatus and the method of processing a capturedimage of the embodiments, the image process section determines whetheran image of an edge portion indicating a contour portion of an object iscontained in the first and second captured images or not, and obtainsthe first mean value and the second mean value while excluding the edgeportion.

Also, in the imaging apparatus and the method of processing a capturedimage of the embodiments, the apparatus further includes an imagingcontrol section which performs imaging by the first pixel group, andimaging by the second pixel group while producing an exposuredifference, and the image process section obtains the level difference,corrects the captured image signals of the pixels of one pixel group ofthe first pixel group and the second pixel group, on the basis of thelevel difference and the exposure difference, and combines the correctedcaptured image signals with the captured image signals of the pixels ofanother pixel group, thereby producing a combined image having a widedynamic range.

Also, in the imaging apparatus and the method of processing a capturedimage of the embodiments, light receiving areas of the pixels of thefirst pixel group, and light receiving areas of the pixels of the secondpixel group are identical with each other within an error range.

Also, in the imaging apparatus and the method of processing a capturedimage of the embodiments, in a full-width at half maximum of spectralsensitivities of colors of the plurality of color filters including thefirst color, a full-width at half maximum of spectral sensitivities ofcorresponding colors of the plurality of color filters including thesecond color is.

Also, in the imaging apparatus and the method of processing a capturedimage of the embodiments, a ratio of a peak value of spectralsensitivities of colors of the plurality of color filters including thefirst color, and a peak value of spectral sensitivities of colors of theplurality of color filters including the second color is within a rangeof 0.9 to 1.1.

Also, in the imaging apparatus and the method of processing a capturedimage of the embodiments, the plurality of color filters including thefirst color are color filters of three primary colors, and the pluralityof color filters including the second color are color filters of thethree primary colors.

Also, in the imaging apparatus and the method of processing a capturedimage of the embodiments, the plurality of color filters configured bythe first color are complementary color filters, and the plurality ofcolor filters configured by the second color are complementary colorfilters.

Also, in the imaging apparatus and the method of processing a capturedimage of the embodiments, a first captured image due to the first pixelgroup is processed to produce a natural color image of an object, and asecond captured image due to the second pixel group is processed toproduce a vivid color image of the object.

Also, a program for processing a captured image of the embodimentscomprises: a step of executing one of the methods of processing thecaptured image.

According to the above-described embodiments, it is possible tosimultaneously capture two kinds of color images having different colortones (for example, a natural color image and a vivid color image), andthe two kinds of color images can be combined with each other withoutproducing a feeling of strangeness to produce a high-definition image oran image having a wide dynamic range.

INDUSTRIAL APPLICABILITY

The imaging apparatus and the like of the invention are equipped with anovel solid-state imaging device, can simultaneously capture two kindsof object color images having different color tones, and can combinethem to produce a high-definition object image or an image having a widedynamic range. Therefore, they are useful when they are applied to awide variety of imaging apparatuses such as a digital still camera, adigital video camera, a camera-equipped electronic device such as, acamera-equipped mobile telephone, a PDA, or a notebook computer, and anendoscope.

Although the invention has been described in detail and with referenceto the specific embodiments, it is obvious to those skilled in the artthat various changes and modifications can be made without departingfrom the spirit and scope of the invention.

The application is based on Japanese Patent Application No. 2010-097367filed Apr. 20, 2010, and its disclosure is incorporated herein byreference.

REFERENCE SIGNS LIST

-   -   10, 20, 30, 40 imaging apparatus    -   11, 22 solid-state imaging device    -   15 CPU    -   17 narrow-spectral pixel mean value calculation circuit    -   18 wide-spectral pixel mean value calculation circuit    -   19, 33, 37 color reproduction selection section    -   20 correction ratio calculation section    -   21 multiplier    -   23 correction ratio suppression section    -   25 narrow pixel edge determination section    -   26 wide pixel edge determination section    -   31 D-range setting device    -   32 exposure difference correction amount calculating device    -   35 adequate-exposure pixel mean value calculation circuit    -   36 underexposure pixel mean value calculation circuit

The invention claimed is:
 1. An imaging apparatus comprising: asolid-state imaging device including: a plurality of pixels which arearranged and formed in a two-dimensional array in a semiconductorsubstrate; a plurality of color filters of a first color which arearranged and stacked in accordance with a predetermined rule on a firstpixel group that includes one of odd rows and even rows of the pixels;and a plurality of color filters of a second color which are arrangedand stacked in accordance with a predetermined rule on a second pixelgroup that includes the other one of the odd rows and the even rows, thecolor filters of the second color being different in spectralsensitivity from the plurality of color filters of the first color; andan image process section which obtains a level difference betweencaptured image signals of the pixels of the first pixel group andcaptured image signals of the pixels of the second pixel group, thelevel difference being due to a spectral sensitivity difference betweenthe plurality of color filters including the first color and theplurality of color filters including the second color, and whichcombines a first captured image that is obtained from the first pixelgroup by correcting the level difference, with a second captured imagethat is obtained from the second pixel group, wherein the image processsection obtains the level difference from a first mean value which is amean value of the captured image signals of a predetermined number ofpixels that are in a periphery of a pixel that is a correction subject,and that belong to the first pixel group, and a second mean value whichis a mean value of the captured image signals of a predetermined numberof pixels that are in a periphery of the pixel that is the correctionsubject, and that belong to the second pixel group, and corrects thelevel difference, and the image process section determines whether animage of an edge portion indicating a contour portion of an object iscontained in the first and second captured images or not, and obtainsthe first mean value and the second mean value while excluding the edgeportion.
 2. The imaging apparatus according to claim 1, wherein theimage process section corrects the level difference by: setting a ratioof the first mean value and the second mean value, as a correctionratio; and multiplying the captured image signal of the pixel that isthe correction subject, by the correction ratio.
 3. The imagingapparatus according to claim 2, wherein when the correction ratioexceeds a preset upper limit value, the image process section uses theupper limit value as the correction ratio.
 4. The imaging apparatusaccording to claim 1, wherein the apparatus further includes an imagingcontrol section which performs imaging by the first pixel group, andimaging by the second pixel group while producing an exposuredifference, and the image process section obtains the level difference,corrects the captured image signals of the pixels of one pixel group ofthe first pixel group and the second pixel group, on the basis of thelevel difference and the exposure difference, and combines the correctedcaptured image signals with the captured image signals of the pixels ofanother pixel group, thereby producing a combined image having a widedynamic range.
 5. The imaging apparatus according to claim 1, whereinlight receiving areas of the pixels of the first pixel group, and lightreceiving areas of the pixels of the second pixel group are identicalwith each other within an error range.
 6. The imaging apparatusaccording to claim 1, wherein in a full-width at half maximum ofspectral sensitivities of colors of the plurality of color filtersincluding the first color, a full-width at half maximum of spectralsensitivities of corresponding colors of the plurality of color filtersincluding the second color is.
 7. The imaging apparatus according toclaim 1, wherein a ratio of a peak value of spectral sensitivities ofcolors of the plurality of color filters including the first color, anda peak value of spectral sensitivities of colors of the plurality ofcolor filters including the second color is within a range of 0.9 to1.1.
 8. The imaging apparatus according to claim 1, wherein theplurality of color filters including the first color are color filtersof three primary colors, and the plurality of color filters includingthe second color are color filters of the three primary colors.
 9. Theimaging apparatus according to claim 1, wherein the plurality of colorfilters configured by the first color are complementary color filters,and the plurality of color filters configured by the second color arecomplementary color filters.
 10. A method of processing a captured imagewhich processes an image captured by a solid-state imaging devicecomprising: a plurality of pixels which are arranged and formed in atwo-dimensional array in a semiconductor substrate; a plurality of colorfilters of a first color which are arranged and stacked in accordancewith a predetermined rule on a first pixel group that includes one ofodd rows and even rows of the pixels; and a plurality of color filtersof a second color which are arranged and stacked in accordance with apredetermined rule on a second pixel group that includes the other oneof the odd rows and the even rows, the color filters of the second colorbeing different in spectral sensitivity from the plurality of colorfilters of the first color, wherein, the method comprise: obtaining alevel difference between captured image signals of the pixels of thefirst pixel group and captured image signals of the pixels of the secondpixel group, the level difference being due to a spectral sensitivitydifference between the plurality of color filters including the firstcolor and the plurality of color filters including the second color;correcting the level difference; and combining a first captured imagethat is obtained from the first pixel group, and a second captured imagethat is obtained from the second pixel group with each other, whereinthe level difference is obtained from a first mean value which is a meanvalue of the captured image signals of a predetermined number of pixelsthat are in a periphery of a pixel that is a correction subject, andthat belong to the first pixel group, and a second mean value which is amean value of the captured image signals of a predetermined number ofpixels that are in a periphery of the pixel that is the correctionsubject, and that belong to the second pixel group, and the leveldifference is corrected, and the method further includes a process ofdetermining whether an image of an edge portion indicating a contourportion of an object is contained in the first and second capturedimages or not, and the first mean value and the second mean value areobtained while excluding the edge portion.
 11. The method of processinga captured image according to claim 10, wherein the level difference iscorrected by: setting a ratio of the first mean value and the secondmean value, as a correction ratio; and multiplying the captured imagesignal of the pixel that is the correction subject, by the correctionratio.
 12. The method of processing a captured image according to claim11, wherein the method further includes, when the correction ratioexceeds a preset upper limit value, using the upper limit value as thecorrection ratio.
 13. The method of processing a captured imageaccording to claim 10, wherein when a captured image obtained byperforming imaging by the first pixel group, and imaging by the secondpixel group while producing an exposure difference is to be processed,the level difference is obtained, the captured image signals of thepixels of one pixel group of the first pixel group and the second pixelgroup are corrected based on the level difference and the exposuredifference, and the corrected captured image signals and the capturedimage signals of the pixels of another pixel group are combined witheach other, thereby producing a combined image having a wide dynamicrange.
 14. The method of processing a captured image according to claim10, wherein light receiving areas of the pixels of the first pixelgroup, and light receiving areas of the pixels of the second pixel groupare identical with each other within an error range.
 15. The method ofprocessing a captured image according to claim 10, wherein in afull-width at half maximum of spectral sensitivities of colors of theplurality of color filters including the first color, a full-width athalf maximum of spectral sensitivities of corresponding colors of theplurality of color filters including the second color is.
 16. The methodof processing a captured image according to claim 10, wherein a ratio ofa peak value of spectral sensitivities of colors of the plurality ofcolor filters including the first color, and a peak value of spectralsensitivities of colors of the plurality of color filters including thesecond color is within a range of 0.9 to 1.1.
 17. The method ofprocessing a captured image according to claim 10, wherein the pluralityof color filters including the first color are color filters of threeprimary colors, and the plurality of color filters including the secondplural colors are color filters of the three primary colors.
 18. Themethod of processing a captured image according to claim 10, wherein theplurality of color filters including the first color are complementarycolor filters, and the plurality of color filters including the secondcolor are complementary color filters.
 19. The method of processing acaptured image according to claim 15, wherein a first captured image dueto the first pixel group is processed to produce a natural color imageof an object, and a second captured image due to the second pixel groupis processed to produce a vivid color image of the object.
 20. Anon-transitory computer-readable recording medium, and which stores aprogram for causing a computer to execute a captured image process forprocessing an image captured by a solid-state imaging device including:a plurality of pixels which are arranged and formed in a two-dimensionalarray in a semiconductor substrate; a plurality of color filters of afirst color which are arranged and stacked in accordance with apredetermined rule on a first pixel group that includes one of odd rowsand even rows of the pixels; and a plurality of color filters of asecond color which are arranged and stacked in accordance with apredetermined rule on a second pixel group that includes the other oneof the odd rows and the even rows, the color filters of the second colorbeing different in spectral sensitivity from the plurality of colorfilters of the first color, wherein, in the image process, obtaining alevel difference between captured image signals of the pixels of thefirst pixel group and captured image signals of the pixels of the secondpixel group, the level difference being due to a spectral sensitivitydifference between the plurality of color filters including the firstcolor and the plurality of color filters including the second color;correcting the level difference; and combining a first captured imagethat is obtained from the first pixel group, and a second captured imagethat is obtained from the second pixel group with each other.